Coding Challenge Submission - #398

This PR implements a Google carousel parser and outputs a JSON list in the expected structure.

Stack

• Ruby 4.x
• rspec
• nokolexbor

Setup

1. Install ruby 4.x using your existing version manager. I used devenv in this case but there's a .ruby-version for asdf/mise/rvm
2. run bundle install
3. run bin/extract to get the output of the provided html file. You can also pass in a specific html file to parse, for example: bin/extract spec/fixtures/pages/mandalorian-cast.html

Use bundle exec rspec to run tests

The approach

I started with using nokolexbor to parse the provided html file using html structure (data-attrid and role="list"/role="listitem") rather than relying on class names for selecting elements. I quickly realized that the thumbnails are set at runtime with Javascript in a <script> tag or lazy loaded from a JSON object once the carousel is scrolled into view.

I chose to extract the images from the JS with regex instead of having a browser or something like QuickJS interpret and execute the code in the script tags. This includes logic to extract lazy loaded thumbnails from a google.ldi map as well. I felt that this was an acceptable tradeoff given the requirements and it wouldn't be difficult to update the code to an approach that executes Javascript if needed.

Once I had an initial implementation, I added rspec tests and started testing with other types of carousels. My initial implementation was overfit on artwork so I changed it to parse for the largest list of single elements containing an image, caption, and link combination. This allowed me to add support for non-artwork types and I decided to remove the artworks key from the expected response.

For general testing strategy, I used a "golden" set of input html files, extracted the JSON array, and compared it against a known good set of resulting JSON files. It reads all files in spec/fixtures/pages and spec/fixtures/expected which makes it easy to add/remove tests over time. I've also added unit tests checking parsing logic/guarantees.

Notes & Tradeoffs

• Google changed the search pages, so a search for "Pablo Picasso Paintings" was not returning the same carousel as the provided sample HTML. I was able to experiment and get a similar carousel for searches like "Mandalorian cast" and "Tom Thomson books".
• Thumbnail images are extracted as base64 encoded data URIs when possible. Lazy loaded images are extracted as gstatic URLs. I return null when I can't find a thumbnail image instead of returning the default placeholder GIF.
• The regex to extract thumbnails from JS is the most brittle part (e.g. if Google changes the data URI pattern for images). I noted the possibility to switch to a browser-based or QuickJS-based approach that executes the page Javascript as a potential future fallback if this becomes an issue.
• I took the liberty of removing the artworks key from the expected results structure and returning the bare JSON array to add support for non-artwork carousel types. I can update it to include the artworks key if it's a requirement.